Correlate two different events to create an incident

#1

Hi,

Is it possible with the Event Orchestration to create an incident based on two (or more) different events that happens in a short period of time?
For exemple, “Create an incident if eventA AND eventB occurs”
If only eventA, i don’t want to create an incident.

It could also work like “Create an incident when receiving eventA AND there is already an incident on service B”

Basically, we’re looking to do some cross service/source correlation

Thanks,
James

1 Like
#2

Hey James,

Thanks for submitting this question on our community forum! Ryan from Support here, and I’m happy to help with this.

By default, Event Orchestration will behave exactly as you have described. I have included a screenshot here to further illustrate:

When writing a rule, simply clicking the “+ And” icon would do what you seem to be describing.

Let us know if you have any other questions, or feel free to submit a support ticket directly at support@pagerduty.com

Cheers,
Ryan

#3

We really don’t have a flexible state/correlation model today that would allow you to model these scenarios. You might get close with the threshold alert rule if you received event A and set a threshold preventing creating the incident for a certain amount of time if you knew that event B will come within that time window. (https://support.pagerduty.com/docs/event-intelligence#threshold-alerts)

#4

Hi Ryan,

Thanks for your help.

I’m not sure this works, i’ve tried this and it seems
“If event.summary matches Test1” AND “if event.summary matches Test2” will try to find Test1 and Test2 in the same event, and not two separate events.

I’m trying to create an incident based on two different alerts/events, one that matches “Test1” and another that matches “Test2”

Doug seems to say in his reply that it’s not possible ?

#5

Unfortunately i don’t know if event B will come

#6

You might need to route those through something first to track state and provide the logic needed to trigger the PD event/alert/incident based on the model you desire. Maybe an AWS Lambda, Step Function, or Event Bridge if you’re an AWS shop. Throwing PD event triggered webhooks might be one way as well but you’d need to quickly reach back into PD via the API to deal with alerts/incidents (assign to a dummy user EP, or quickly resolve) while waiting for event B or “resetting” after some default period.

#7

Hello,
My 2 cents on this. We have sometimes the same kind of use cases.

eg: no incident (or low urgency incident) when 1 on 3 network links is down (ie: 1st event is “network link A down”) but trigger an incident (or set high incident) when 2 on 3 network links are down (ie: 2nd event is “network link B down”)

Currently we’re able to solve this when we use some monitoring tools (eg: Datadog or Google Cloud Monitoring lets you create this kind of AND conditions with monitors). But not with others.

Regards

#8

Indeed, Datadog monitors let us do some composite monitors that can solve this use-case.
But I’d love to be able to do this regardless of the monitoring source in pagerDuty

#9

I agree. If I was not clear, it’s why I wrote:

But not with others.

1 Like
#10

Just to update this thread as I had some answers from PagerDuty.
There is a new Global Orchestration coming up soon and i was able to test it and confirm it can solve my use case.